Skip to content

analyze: deduplicated size of a set of archives, #5741 - #9971

Merged
ThomasWaldmann merged 2 commits into
borgbackup:masterfrom
ThomasWaldmann:analyze-set-dedup-5741
Jul 30, 2026
Merged

analyze: deduplicated size of a set of archives, #5741#9971
ThomasWaldmann merged 2 commits into
borgbackup:masterfrom
ThomasWaldmann:analyze-set-dedup-5741

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Jul 29, 2026

Copy link
Copy Markdown
Member

Implements the request in #5741: report the deduplicated size of a set of archives (e.g. all backups matching -a 'sh:userA-*'), following ThomasWaldmann's proposal there.

Two commits: the analyze feature, plus an unrelated one-liner fixing a stale borg info epilog.

Every figure is reported as source (uncompressed source data) and stored (compressed, as stored in the repository) size, plus the compression factor relating the two (stored / source).

Deduplicated size of a set of archives

For the considered (matching) set of archives:

  • Deduplicated size of set — summed size of the union of chunks the set references (chunks shared within the set counted once).
  • Exclusive size of set — summed size of chunks referenced only by the set and by no other archive, i.e. the space that deleting the whole set would free.
  • Unreferenced chunks — chunks that no non-deleted archive references, i.e. what borg compact could free in the current state of the repository. Only their stored size is known: a chunk's source size is only recorded in the archives referencing it, and these have none.

Following the issue comment, chunks are flagged as referenced by the considered set (C) and/or by the rest of the archives (R); the exclusive size is sum(size(x) for x in C - R).

$ borg analyze -a 'sh:web'

Deduplicated size of the 2 considered archive(s)
===================================================================
Considered archives: 2 (of 4 in the repository)

                                  source        stored  compression
Deduplicated size of set:        8.00 MB       3.26 MB         0.41
Exclusive size of set:           2.00 MB     820.62 kB         0.41
Unreferenced chunks:                 n/a           0 B          n/a

Without an archive filter, the considered set is all archives - every referenced chunk is then trivially exclusive to it, so that line is suppressed and the report is labeled as the whole repository.

Decomposition by archive name (--by-name)

Archives sharing a name form a series, so a name usually groups all backups of one source; old-style archives that do not form a series have one name each and are grouped just as well.

Every chunk is counted in exactly one row, so the rows add up to the repository's deduplicated size. This answers "which name costs how much, and what would I get back by dropping it" for all names at once:

$ borg analyze --by-name

Repository decomposition by archive name
=================================================================================
4 archive(s) with 3 distinct name(s)

name                            archives        source        stored  compression
db                                     1       4.00 MB       1.64 MB         0.41
logs                                   1       3.00 MB       1.22 MB         0.41
web                                    2       2.00 MB     820.62 kB         0.41
(shared by 2+ names)                           6.00 MB       2.44 MB         0.41
(unreferenced)                                     n/a           0 B          n/a
---------------------------------------------------------------------------------
total (deduplicated)                   4      15.01 MB       6.11 MB         0.41

This needs only a single pass over the archives: bits 5..22 of the ephemeral user flags hold the archive name that first referenced a chunk (as index + 1, so 0 means unreferenced), bit 23 marks chunks referenced by more than one name. As the shared and unreferenced rows can only be determined by looking at every archive, --by-name always covers the whole repository and cannot be combined with archive filters.

Implementation

  • Reuses the repository's own chunk index (repository.chunks) instead of building a second index — it already holds each chunk's stored size (obj_size). The membership flag bits are OR-ed in and the source size (0 in the repo index) is filled in from the per-archive references cache, preserving pack_id/obj_offset/obj_size via _replace. These in-memory mutations never persist: write_chunkindex_to_repo zeroes flags and size, and close() only serializes F_NEW entries (none here).
  • Chunk membership and source sizes come from the per-archive references cache maintained by borg compact, so unchanged archives usually do not need to be opened/decrypted.
  • The references-cache helpers (get_archive_references & friends, ArchiveReferences) are moved from archiver/compact_cmd.py into cache.py as free functions so analyze and compact share one implementation; compact keeps its previous behavior.

Notes

  • The dedup-size report works for one or more matching archives; the existing directory hot-spot report still requires at least two (skipped with an info log otherwise).
  • -a needs the sh: prefix for globs (borg2's default match is exact).
  • Chunks of soft-deleted archives count as unreferenced (soft-delete keeps the archive's chunks and its reference cache; borg compact only keeps them when the repository is damaged, so borg undelete stays possible).
  • Rows whose chunks carry no source size at all (e.g. only archive metadata, which is recorded with size 0) show n/a in the compression column rather than dividing by zero.

Tests

analyze_cmd_test.py covers: set dedup vs exclusive size with shared and unique files, the single-archive set, the whole-repository (no filter) report, the --by-name decomposition (asserting the rows add up to the total), the rejection of filters with --by-name, and the unreferenced-chunks report - including that borg compact then frees exactly the reported number of objects.

Rebased onto current master; analyze/compact/cache/info suites pass there (94 tests), mypy reports no new errors.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.84462% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.83%. Comparing base (8b5c78b) to head (28de45c).
⚠️ Report is 60 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/archiver/analyze_cmd.py 90.28% 10 Missing and 7 partials ⚠️
src/borg/cache.py 84.50% 9 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9971      +/-   ##
==========================================
+ Coverage   85.77%   85.83%   +0.05%     
==========================================
  Files          95       95              
  Lines       16869    17034     +165     
  Branches     2582     2607      +25     
==========================================
+ Hits        14470    14621     +151     
- Misses       1665     1673       +8     
- Partials      734      740       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Comment thread src/borg/archiver/analyze_cmd.py Outdated
Comment thread src/borg/archiver/analyze_cmd.py Outdated
ThomasWaldmann and others added 2 commits July 30, 2026 15:11
`borg analyze` now reports how much space a set of archives occupies and how much
deleting it would actually free. All figures come as source (uncompressed source
data) and stored (compressed, as stored in the repository) size, plus the
compression factor relating the two.

For the considered (matching) set of archives:

- deduplicated size of the set: the summed size of the union of chunks the set
  references (chunks shared within the set counted once),
- exclusive size of the set: the summed size of chunks referenced only by the set
  and by no other archive, i.e. what deleting the whole set would free,
- unreferenced chunks: chunks no non-deleted archive references, i.e. what
  `borg compact` could free in the current state of the repository.

Following ThomasWaldmann's proposal in borgbackup#5741, chunks are flagged as referenced by
the considered set (C) and/or by the rest of the archives (R); the exclusive size
is sum(size(x) for x in C - R). Without an archive filter the considered set is all
archives, so everything referenced is trivially exclusive to it - the report is then
labelled as covering the whole repository and that line is left out.

`--by-name` decomposes the whole repository by archive name instead: one row per
name with what is exclusive to it, one row for the chunks shared by 2+ names and one
for the unreferenced ones. Every chunk is counted in exactly one row, so the rows
add up to the repository's deduplicated size. Archives sharing a name form a series,
so a name usually groups all backups of one source; old-style archives that do not
form a series have one name each and group just as well. This needs a single pass:
flag bits 5..22 hold the name that first referenced a chunk (as index + 1, so 0 means
unreferenced) and bit 23 marks chunks referenced by more than one name. As the shared
and unreferenced rows require looking at every archive, --by-name always covers the
whole repository and rejects archive filters.

Rather than building a second index, this reuses the repository's own chunk index:
it already holds every chunk's stored size (obj_size). The flag bits are OR-ed in and
the source size (0 in the repo index, it is only recorded in the archives referencing
a chunk) is filled in from the per-archive references cache, keeping the pack location
intact via _replace. The mutations never persist: write_chunkindex_to_repo zeroes flags
and size, and close() only serializes F_NEW entries, of which there are none here.

Chunk membership and source sizes come from the per-archive references cache that
`borg compact` maintains, so unchanged archives usually need not be opened at all.
Its helpers (get_archive_references and friends, ArchiveReferences) move from
archiver/compact_cmd.py to cache.py as free functions, so analyze and compact share
one implementation; compact keeps its previous behaviour.

The pre-existing directory hot-spot report is unchanged and still needs 2+ archives.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
borg info no longer computes per-archive deduplicated sizes (Archive.calc_stats
sets usize=0 as it is expensive), and the output only shows the original size.
The epilog still explained "this archive vs all archives deduplicated size",
which no longer matches what the command prints.

Replace it with a note that deduplicated sizes are not shown here and point to
`borg analyze` (deduplicated size of a set of archives) and `borg compact --stats`
(repository-wide deduplicated size).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann force-pushed the analyze-set-dedup-5741 branch from bca673a to 28de45c Compare July 30, 2026 13:19
@ThomasWaldmann
ThomasWaldmann merged commit 13a62e7 into borgbackup:master Jul 30, 2026
18 of 19 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the analyze-set-dedup-5741 branch July 30, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant